Motivation

The server is launched with HF_HUB_OFFLINE=1 and is meant to treat model files as read-only; however, the fast tokenizer conversion happening in the launcher does not follow this (if a revision is not passed). This can cause problems if a model in HF Hub is updated and the tokenizer conversion downloads the tokenizer files for the new commit of the model but then the server doesn't download the new model files... the server fails to load because it can't find the model files.

Modifications

• Set local_files_only=True with and without the revision arg when doing the fast tokenizer conversion
• Set HF_HUB_OFFLINE=1 in the env as well for good measure
• Little refactoring to have the command building be shared

Result

Fast tokenizer conversion in the launcher should never download new files.

Related Issues

• Fast tokenizer conversion added in Update launcher to auto-convert fast tokenizer #48
• Setting local_files_only if revision is passed: 🐛 disallow downloads for fast tokenizer conversion #63